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Introduction 

Expert  judgment  In  any  field  Is  thought  to  be  a  careful,  reliable  process 
based  on  a  formal  education  and  developed  through  years  of  experience. 
Clashes  of  opinion  among  experts  are  therefore  thought  to  re^  isent 
differences  In  traditions,  conflicts  between  world  views  that  are  presumed  to 
be  coherent  and  stable.  This  assumption  Is  pervasive.  For  example.  It 
underlies  the  use  of  a  "hit  list"  by  Environmental  Protection  Agency 
adnlnlstrators  to  exclude  scientists  from  participating  In  review  panels  on 
the  basis  of  their  presumed  judgment  policies  (Marshall,  1982).  But  is  It 
true? 

If  experts  were  to  make  the  same  substantive  judgment  about  the  same  set 
of  objects  under  different  task  conditions,  would  they  exhibit  stable 
Individual  styles?  Would  the  application  of  their  expert  opinion  Involve  the 
same  judgnent  policy  despite  variation  In  the  details  of  the  task? 

The  comnon  view  would  predict  that  differences  among  Individual  experts 
would  be  stable  enough  to  withstand  the  Influence  of  variations  In  task 
condition.  This  assumption  underlies  the  discussion  of  the  Importance  of 
Individual  styles  In  the  design  of  decision  support  systems  reviewed  by  Sage' 
(1981).  Another  view  would  argue  that  the  Influence  of  the  task  conditions 
upon  the  cognitive  processes  Involved  In  the  judgments  may  be  large  enough  to 
override  differences  In  Individual  judgment  policies  (Brunswlk,  1956; 
Hammond,  1955).  It  has  been  shown,  for  example,  that  college  students' 
judgments  may  be  Influenced  by  details  of  the  task  (e.g.,  response  mode, 
Lichtenstein  A  Slovlc,  1971),  and  that  experts'  performance  of  statistical 
Inference  on  abstract  word  problems  may  deviate  systematically  from  the  norm 
due  to  use  of  heuristic  shortcuts  (Elnhorn  A  Hogarth,  1981;  Kahneman,  Slovlc, 


Page  2 


8  Tversky,  1982 ) ^ 

The  study  of  highway  experts'  judgments  of  safety  reported  In  Hammond, 
Haim,  Grassla,  and  Pearson  (1983)  provided  an  opportunity  to  test  the 
stability  of  Individual  judgment  policies  In  expert  decision  makers.  In  that 
study  21  highway  engineers  from  government,  private  Industry,  and  university 
settings  judged  the  safety  of  39  highways  under  three  conditions.  Each 
condition  was  designed  to  Induce  either  Intuitive,  quasl-ratlonal ,  or 
analytical  cognition.  The  experts  devoted  much  time  to  their  judgments,  and 
the  task  characteristics  Involved  a  variety  of  judgment  conditions  that  are 
Involved  In  an  engineering  environment.  Since  each  expert  judged  the  same  set 
of  highways  In  each  condition  It  Is  possible  to  address  the  following 
questions:  Did  an  expert  use  the  same  judgment  policy  In  each  task  condition? 
More  specifically,  were  an  expert's  judgments  of  the  safety  of  the  highways  In 
one  task  condition  more  similar  to  his  judgments  In  the  other  task  conditions 
than  they  were  to  the  judgments  of  the  other  experts  In  the  same  task 
condition?  Two  approaches  were  used  to  explore  these  questions:  examination 
of  correlations  among  performances  and  cluster  analysis  of  performances. 

First  approach:  Correlations  among  Performances  f 

The  average  Intercorrelatlons  between  the  different  Individuals' 
performances  on  the  same  task  condition  and  between  the  same  Individual's 
performances  on  the  different  task  conditions  are  presented  In  Table  1.  The 
mean  Intra-task  correlation  Is  .56  and  the  mean  Intra- Individual  correlation 
Is  .50.  (These  may  be  compared  with  the  mean  correlation  of  .50  between  the 
expert's  judgment  and  the  true  answers.)  Although  the  mean  Intra- task 
similarity  Is  higher  than  the  mean  Intra- Individual  similarity,  there  Is  much 
overlap.  Further,  the  wide  variation  In  the  correlation  scores  suggests  that 


for  some  Individuals  similarities  due  to  task  condition  are  greater  than 
similarities  due  to  stability  of  Individual  judgment  policy,  whereas  for 
others  the  reverse  Is  truel  Thus,  this  analysis  of  correlations  among 
performances  gives  equivocal  evidence  that  task  conditions  have  a  slightly 
stronger  role  In  determining  similarities  among  experts'  performances  than 
stable  Individual  differences  have. 

Insert  Table  1  about  here 


Second  approach:  Cluster  Analyses  of  Performances 

Cluster  analysis  makes  It  possible  to  determine  whether  groups  of  similar 
performances  are  made  up  of  performances  by  the  same  Individuals  over 
different  task  conditions,  or  of  performances  by  dl fferent  Individuals  under 
the  same  task  conditions.  If  each  cluster  produced  by  the  analysis  consisted 
of  the  performances  of  a  different  subset  of  Individuals  over  all  three 
conditions,  we  would  conclude  that  each  cluster  represented  a  group  of  experts 
with  stable  judgment  policies,  and  that  Individual  judgment  policies  are 
robust  under  variation  of  task  conditions.  On  the  other  hand.  If  each  cluster 
consisted  of  the  performances  In  one  particular  condition  by  all  Individuals/ 
we  would  conclude  that  specific  task  conditions  dominate  Individual 
differences  In  determining  judgment  policy  when  experts  judge  highway  safety. 


Page  4 


Method 


Performance 

An  expert's  performance  In  judging  safety  under  the  Intuitive  or 
quasi -rational  condition  consisted  of  the  judgments  he  made  for  each  of  39 
highways.  In  the  analytical  condition  the  expert  composed  a  formula  for 
determining  highway  safety,  and  performance  consists  of  the  answers  produced 
by  applying  the  formula  to  each  highway.  Since  7  of  the  21  experts  made 
errors  In  constructing  their  formulas  which  would  have  produced  unintended 
answers  and  thus  would  have  obscured  the  stability  of  their  Individual 
approaches  to  judging  highway  safety,  these  errors  were  corrected. 

Description  of  Judgment  Policy 

Ten  dimensions  pertaining  to  safety,  such  as  lane  width  and  number  of 
curves  per  mile,  were  measured  for  each  highway  Independently  of  the  experts' 
judgments.  These  dimensions  were  used  to  made  a  linear  model  of  the  expert's 
judgments  (or  of  the  answers  generated  from  his  formula)  under  each  task 
condition.  The  beta  weights  from  these  models  Indicate  the  relative  weight  or 
Importance  the  expert  placed  on  each  dimension  under  that  condition.  , 

Beta  weights  are  commonly  used  to  describe  judgment  policies.  However, 
they  contain  Information  only  about  the  linearly  predictable  portions  of  a  set 
of  judgments.  To  guard  against  possible  loss  of  Information  about  systematic 
nonlinear  use  of  cues,  both  the  experts'  beta  weights  and  the  answers 
themselves  were  used  as  descriptions  of  the  experts'  judgment  policies  for 
cluster  analysis.  Thus,  cluster  analyses  were  performed  In  parallel  on 
profiles  representing  the  experts'  judgments  of  the  highway  (39  dimensions) 
and  profiles  representing  their  beta  weights  from  the  linear  model  of  those 


judgments  (10  dimensions)*.  There  were  62  profiles  of  judgments  of  highways,  3 
for  each  of  21  experts  performing  in  each  condition  (intuitive, 
quasi • rati onal ,  and  analytical),  with  the  exception  of  one  expert  whose 
Intuitive  answers  were  Incomplete.  There  were  63  profiles  of  beta  weights  for 
the  second  cluster  analysis,  since  the  expert  with  Incomplete  data  was 
Included,  enough  of  his  answers  being  available  to  allow  construction  of  a 
regression  model. 

The  cluster  analyses  were  carried  out  with  the  BMDP2M  program  (Dixon, 
Brown,  Engelman,  Frane,  &  Jennrlch,  1977).  The  measure  of  similarity  was  the 
Intercorrelation  between  profiles.  From  the  tree  display  output  of  this 
program,  clusters  were  identified  according  to  the  following  criteria: 

1.  Each  cluster  had  to  have  at  least  10  members. 

2.  Two  clusters  with  ten  or  more  members  each  would  not  be  combined,  but 
rather  would  be  considered  distinct  clusters. 

3.  Every  Item  that  did  not  fall  Into  a  cluster  Identified  according  to  these 
criteria  was  swept  Into  the  class  of  "outliers". 

1 

Results 

The  composition  of  the  clusters  was  examined  with  respect  to  the  number 
of  profiles  In  each  that  were  produced  under  the  same  task  conditions  or  by 
the  same  experts. 


The  Role  of  Task  Condition  In  Determining  Clusters 


1.  Clusters  of  highway  judgment  profiles. 

Three  clusters  and  13  outliers  were  Identified  In  the  highway 
judgment  profiles.  Table  2  shows  the  number  of  profiles  In  each  cluster, 
the  number  of  each  task  condition  that  would  be  expected  by  chance  If  task 
condition  were  not  related  to  cluster  membership,  and  the  number  that 
actually  appeared.  A  chi-squared  test  of  each  cluster  shows  that  the 
distribution  of  profiles  Is  significantly  different  from  chance,  and, 
therefore,  task  conditions  had  a  determining  role  In  producing  the 
clusters.  The  task  distribution  for  the  set  of  outliers  Is  not 
significantly  different  from  chance. 

Insert  Table  2  About  Here 


2.  Clusters  of  beta  weight  profiles. 

Two  clusters  and  4  outliers  were  Identified  In  the  beta  weight 
profiles.  Table  3  shows  the  number  of  profiles  In  each  cluster  and  the' 
number  of  each  task  condition  that  would  be  expected  and  that  was 
observed.  Chi-squared  analysis  of  these  data  also  show  that  task  had  a 
determining  role  In  producing  the  clusters.  The  set  of  outliers  was 
tested  by  the  procedure  outlined  In  the  appendix,  and  was  not  significant. 


Insert  Table  3  About  Here 


The  Role  of  Stable  Individual  D1 f ferences  In  Determining  Clusters 

Although  the  above  analysis  showed  that  the  task  condition  played  a 
significant  role  In  determining  cluster  membership,  stable  Individual 
differences  might  also  have  played  such  a  role.  That  possibility  was  tested 
for  each  cluster  analysis  by  determining  whether  performances  of  Individuals 
were  concentrated  significantly  In  single  clusters. 

1.  Clusters  of  highway  judgment  profiles. 

Table  4  shows  that  there  were  a  number  of  experts  for  whom  the 
profiles  of  judgments  under  two  or  three  different  task  conditions  were 
similar  enough  to  appear  In  the  same  cluster:  two  In  Cluster  HJ-2 
(experts  4  and  20)  and  three  In  Cluster  HJ-3  (experts  5,  10,  and 
17— Including  all  three  of  expert  5's  performances).  Additionally,  three 
experts  (3,  9,  and  13)  had  two  performances  that  were  outliers. 

Insert  Table  4  About  Here 


As  some  such  clustering  of  Individuals  would  be  expected  to  occur  by 
chance,  a  procedure  for  determining  the  probability  of  the  observed 
distribution  pattern  was  applied  (see  Appendix).  The  even  distribution 
across  Individuals  In  clusters  HJ-1  and  HJ-2  was  marginally  significant,  a 
result  that  Is  the  opposite  from  what  would  be  expected  If  stable 
Individual  differences  In  experts'  judgment  policies  determined  the 
pattern  of  similarities  among  their  judgments. 
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2!  Clusters  of  beta  weight  profiles! 

The  data  from  the  two  clusters  and  the  set  of  outliers  are  shown  in 
Table  5.  As  there  were  three  tasks  per  expert  and  only  two  clusters, 
there  must  be  pairs  from  the  same  expert  in  one  or  the  other  cluster. 
However,  the  experts  were  distributed  significantly  more  evenly  among  the 
clusters  than  would  be  expected  by  chance. 

Insert  Table  5  About  Here 


In  summary,  the  cluster  analysis  method  shows  strong  evidence  that  task 
conditions,  rather  than  stable  Individual  differences,  determine  similarities 
among  judgment  performances  and  judgment  policies. 

Discussion 

The  cluster  analysis  approach  offers  more  Insight  Into  the  relative 
Importance  of  task  conditions  and  stable  Individual  judgment  policy  In 
determining  experts'  judgment  performances  than  does  the  approach  of  comparing 
wlthln-task  and  within- Individual  correlations.  Comparing  the  distributions 
of  correlations  reveals  that  the  intra-task  correlations  are  slightly  higher/ 
on  the  average,  than  the  Intra- Individual  correlations,  but  the  variability  is 
so  high  that  Interpretation  Is  difficult.  Making  further  use  of  these 
correlations  to  produce  clusters  of  similar  performances  allows  us  to  ask 
whether  membership  In  these  clusters  Is  determined  by  task,  by  Individual,  or 
by  both.  It  was  clearly  task  conditions,  and  not  Individual  style,  that 
determined  cluster  membership.  Thus  the  cluster  analysis  approach  offers  an 
unambiguous  answer  to  the  question,  while  the  correlation  approach  does  not. 


It  Is  curious  that  the  cluster  analysis  of  the  safety  judgments  of  the  39 
highways  produces  three  distinct  clusters,  whereas  the  cluster  analysis  of  the 
beta  weights  on  the  10  dimensions  In  the  linear  models  of  the  Individual 

experts'  judgments  produces  only  two  clusters.  Inspection  of  Tables  2  and  3 
shows  the  nature  of  those  clusters.  With  the  judgments,  one  cluster  Involves 
mainly  judgments  produced  under  the  Intuitive  task  conditions,  another  mainly 
quasi- rational  judgments,  and  the  third  mainly  analytic  judgments.  With  the 
beta  weights,  one  cluster  Involves  mainly  the  linear  models  of  judgments  under 
the  Intuitive  task  conditions  and  the  other  mainly  models  of  both 

quasl-ratlonal  and  analytic  judgments. 

The  judgments  produced  by  the  quasl-ratlonal  and  analytical  tasks  are 

more  distinct  from  each  other  than  are  the  linear  model  descriptions  of  these 
same  judgments;  that  Is,  distinct  patterns  of  judgments  are  fit  by  linear 

models  that  are  not  distinct.  How  Is  this  to  be  explained?  There  may  be 
something  in  common  among  the  nonlinear  portions  of  the  quasl-ratlonal 
judgment  policies  that  Is  distinct  from  the  nonlinear  portions  of  the  analytic 
judgment  policies  (formulas).  This  could  cause  the  quasl-ratlonal  and 
analytical  performances  to  be  grouped  separately  In  the  cluster  analysis  of 
the  highway  judgments  (which  contain  both  linear  and  nonTTnear  portions  of  the* 
judgments  as  well  as  noise),  but  to  be  grouped  together  In  the  cluster 
analysis  of  the  beta  weight  data,  where  the  distinct  nonlinear  portions  of  the 
judgment  strategies  have  been  excluded. 


Conclusion 


In  this  study  the  task  characteristics  a  stronger  role  than  stable 
Individual  judgment  policies  In  determining  similarity  among  experts' 
performances  In  a  highway  safety  judgment  task.  The  results  support  the  view 
that  the  task  conditions  under  which  expert  judgments  are  made  play  a  very 
Important  role  In  determining  how  those  judjnents  are  made,  and  consequently 
In  determining  the  judgments  themselves. 
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Table  1 

Average  Intercorrelations  Within  Task  and  Within  Individual 


Correlations  among  Different  Individuals  on  Each  Task  Condition*: 


Lowest 

Mean 

Highest 

N 

Intuitive 

.06 

.58 

.86 

20 

Quasl-ratlonal 

-.20 

.47 

.82 

21 

Analytic 

-.18 

.63 

.93 

21 

Mean  within  task  condition 
correlation  for  all  three 
task  conditions:  .56 


Correlations  among  different  task  conditions  (I  *  Intuitive, 


Q  *  quasi - rati onal ,  A  * 

analytical)  for 

the  same  Individual: 

Lowest 

Mean 

Highest 

N 

r(I,Q) 

-.01 

.43 

.67 

20 

r(I,A) 

-.01 

.49 

.66 

20 

r(Q.A) 

.06 

.56 

.87 

21 

Mean  of  the 

three  r's  for 

each  Individual 

:  .06 

.50 

.68 

21 

Correlations  between  Individual 

judgments  of  highway 

safety  and  the  true 

safety  of  the  highways. 

for  eacti 

i  task  condition. 

Intuitive  Quasl-ratlonal  Analytical 

Mean 

.53 

.48 

.49 

.50 

*  The  mean  correlation  for  the  analytical  task  condition  here  Is  different 
from  that  reported  In  Hamnond,  Hamm,  Grassla,  and  Pearson  (1983)  because 
here  we  treat  errors  In  the  analytical  formulas  liberally,  while  there  we 
treated  them  conservatively. 


Clusters  of  Highway  Judgments 


The  expected  and  observed  numbers  of  performances  In  each  task  condition 
that  fell  Into  each  of  the  three  clusters  of  highway  judgments. 


Table  3 

Clusters  of  Beta  Weights 


The  expected  and  observed  numbers  of  performances  In  each  task  condition 
that  fell  Into  each  of  the  two  clusters  of  beta  weights. 


Cluster 

N 

Task 

Expected 

Observed 

Chi -squared 

Significance 

BW-1 

26 

I 

8.67 

21 

27.77 

p  <  .001 

Q 

8.67 

0 

A 

8.67 

5 

BW-2 

33 

I 

11 

0 

18.73 

p  <  .001 

Q 

11 

20 

A 

11 

13 

Outliers 

4 

I 

1.33 

0 

NS,  p  <  .32' 

Q 

1.33 

1 

A 

1.33 

3 

*  Exact  probability  (see  Appendix). 


Table  4 

Distribution  of  Individuals  In  Clusters,  Highway  Judgments 


The  expected  and  observed  numbers  of  performances  by  each  expert  that  fell 
Into  each  of  the  three  clusters  of  highway  judgments. 


Cluster 


HJ-1 

HJ-2 

HJ-3 

Outliers 

N 

12 

18 

19 

13 

Expected 

.60 

.86 

.9i 

.62 

Observed: 
Expert  # 

1* 

0 

1 

0 

1 

2 

0 

1 

1 

1 

3 

0 

0 

1 

2 

4 

1 

2 

0 

0 

5 

0 

0 

3 

0 

6 

1 

0 

1 

1 

7 

1 

1 

1 

0 

8 

1 

1 

1 

0 

9 

0 

1 

0 

2 

10 

0 

0 

2 

1 

11 

1 

1 

1 

0 

12 

1 

1 

1 

0 

13 

0 

0 

1 

2 

15 

1 

1 

1 

0 

17 

0 

1 

2 

0 

19 

1 

1 

0 

1 

20 

1 

2 

0 

0 

21 

1 

1 

1 

0 

22 

1 

1 

1 

0 

23 

0 

1 

1 

1 

24 

1 

1 

0 

1 

Probability  of  observing  a  pattern: 

This  concentrated  or  more: 

1.0  .995  .716  .440 

This  concentrated  or  less: 

.059  .046  .342  .799 


♦Expert  1  did  not  have  an  Intuitive  performance. 


Table  5 

Distribution  of  Individuals  In  Clusters,  Beta  Weights 


The  expected  and  observed  numbers  of  performances  by  each  expert  that  fell 
Into  each  of  the  two  clusters  of  Beta  Weights. 


Cluster 


N 

BW-1 

26 

BW-2 

33 

Outliers 

4 

Expected 

1.24 

1.57 

0.19 

Observed: 
Expert  # 

1 

2 

1 

0 

2 

1 

2 

0 

3 

1 

1 

1 

4 

2 

1 

0 

5 

1 

2 

0 

6 

1 

2 

0 

7 

1 

2 

0 

8 

2 

1 

0 

9 

1 

1 

1 

10 

1 

2 

0 

11 

2 

1 

0 

12 

1 

2 

0 

13 

1 

1 

1 

15 

2 

1 

0 

17 

1 

1 

1 

19 

1 

2 

0 

20 

1 

2 

0 

21 

1 

2 

0 

22 

1 

2 

0 

23 

1 

2 

0 

24 

1 

2 

0 

Probability  of  observing  a  cluster: 

This  concentrated  or  more: 

1.0 

1.0 

1.0 

This  concentrated  or  less: 


A  method  has  been  developed  and  computerized  for  determining  the 
probability  that  .1  sample  from  a  universe  of  C  categories,  E  exemplars  per 
category,  should  have  any  particular  distribution  of  exemplars  over 
categories.  As  It  Is  possible  to  order  the  possible  distributions  according 
to  the  degree  that  the  exemplars  are  concentrated  In  a  few  categories,  the 
method  also  calculates  the  probability  that  an  observed  distribution  could 
have  been  this  concentrated  or  more  (or  less)  by  chance. 

This  method  Is  useful,  as  In  the  present  study,  for  determining  whether  a 
cluster  of  performances  has  a  disproportionately  high  or  low  number  of 
performances  by  the  same  Individuals.  More  generally.  It  Is  useful  for 
determining  whether  a  distribution  of  objects  over  a  group  of  Individuals  Is 
significantly  "unfair",  l.e.,  more  concentrated  than  would  be  expected  by 
chance.  In  effect,  the  method  provides  a  substitute  for  the  Chi-Squared  test 
with  equal  expected  frequencies.  It  gives  the  exact  probability  of  the 
observed  distribution  even  when  the  expected  frequency  per  category  Is  less 
than  5.  (If  the  expected  frequency  per  category  Is  much  greater  than  5,^ 
however,  chi- squared  1$  preferable  to  the  present  method  because  It  Is 
computationally  quicker.) 

In  the  present  study  there  was  a  universe  of  21  experts  (categories),  3 
performances  (exemplars)  each.  Clusters  of  size  N  were  produced.  Were  the 
experts'  Individual  Identities  Influential  In  determining  cluster  membership? 
The  observed  distribution  of  performances  over  experts  In  the  cluster  may  be 


represented  as  a  row  of  numbers  r^  ranging  from  rQ  to  r3. 

Table  A-l 

Index,  i  0123 

Cell  value,  r.j  rQ  rj  r ^  r3 

Very  concentrated 

example  pattern  for  a  17  0  0  4 

12- performance  cluster 

Very  non- concentrated 

example  pattern  for  a  9  12  0  0 

12-performance  cluster 

The  indices  1  represent  the  number  of  performances  an  expert  could  have  in  a 
given  cluster.  The  cell  values,  r^'s,  represent  the  number  of  experts  who  had 
1  performances  In  the  cluster.  The  sum  of  the  r^'s  equals  21,  the  total 
number  of  experts.  The  sum  of  (i  times  r^'s  equals  the  total  number  of 
performances  in  the  cluster. 

To  determine  how  often  the  observed  pattern  could  occur  as  opposed  to 
other  possible  patterns.  It  Is  necessary  to  consider  the  other  possible 
patterns  as  well.  The  method  Involves  these  steps:  (a)  listing  each  possible 
pattern  as  a  row  In  a  table,  (b)  arranging  the  rows  In  decreasing  order  of 
concentratedness,  (c)  calculating  the  number  of  possible  ways  that  each* 
pattern  could  occur,  (d)  determining  the  position  of  the  row  that  represents 
the  observed  distribution,  and  finally  (e)  summing  over  the  rows  up  through 
the  row  representing  the  observed  distribution.  In  order  to  determine  the 
likelihood  that  a  distribution  this  concentrated  or  more  (or  less)  would 
occur.  We  demonstrate  the  method  by  considering  a  cluster  of  size  4. 


Step  U  Producing  the  table! 


There  are  4  columns  In  the  table ,  corresponding  to  an  Individual  having  1 
■  0,  1,  2,  or  3  performances  In  the  cluster.  The  column  entry,  r^,  Indicates 
the  number  of  Individuals  who  had  1  performances  In  the  cluster.  Each  row  of 
the  table  represents  a  different  pattern  of  distributions  of  4  performances 
over  Individual  experts.  Thus  the  first  row  represents  a  cluster  with  high 
concentration  of  performances  on  a  few  Individuals:  one  Individual  had  three 
performances  and  another  Individual  had  the  fourth.  The  second  row  represents 
a  cluster  made  up  of  two  performances  from  each  of  two  experts.  The  fourth 
row  has  the  lowest  concentration  pattern,  one  performance  from  each  of  4 
experts. 

Table  A-2 

1  *  Number  of  performances  by  an  Individual. 

0  12  3 

Row  1  19  1  0  1 

Row  2  19  0  2  0 

Row  3  18  2  1  0 

Row  4  17  4  0  0 

» 

Step  2.  Ordering  the  rows. 

The  rows  must  be  arranged  In  order  of  concentratedness.  An  Index  of 
concentratedness  Is  provided  by  adding  how  many  performances  were  In  the 
cluster  from  the  j  most-concentrated  experts,  as  j  goes  from  1  to  21  (the 
number  of  experts)  or  to  4  (the  number  of  performances  In  the  cluster), 
whichever  Is  smaller.  Generally,  the  formula  for  calculating  the  Index  of 
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concentratedness  for  a  row  Is: 

N 

l  (r4  for  the  j  most-concentrated  experts) 

j-1  1 

where  N  *  the  number  of  experts  or  number  of  performances,  whichever  Is  less. 


Table  A-3 


Demonstration  of  the  procedure  for  calculating,  From  Table  A-l,  the  number  of 
performances  by  the  j  most-concentrated  experts,  as  j  varies  from  1  to  4. 

j 

1  2  3  4  Sum 


Row  1 
Row  2 
Row  3 
Row  4 


3  4  4  4  15 
2  4  4  4  14 
2  3  4  4  13 
1  2  3  4  10 


Comparing  the  Indices  of  concentratedness  for  Rows  1  to  4,  In  the  rightmost 
column  of  Table  A-3,  shows  that  the  rows  In  Table  A-2  are  In  the  required 
order.  Otherwise  they  would  need  to  be  rearranged  In  order  of  decreasing 
concentratedness . 

Step  3.  Calculating  the  number  of  ways  that  each  row  cquld  happen. 

i 

Table  A-2  simply  enumerates  the  possible  patterns  of  distribution  of 
performances  over  experts  for  clusters  of  size  4,  regardless  of  the  Individual 
Identities  of  the  experts  with  each  number  of  performances.  In  order  to 
calculate  the  probability  of  getting  the  pattern  represented  by  a  row,  the 
number  of  ways  the  row's  pattern  could  happen  must  be  calculated. 


This  calculation  requires  addressing  two  questions: 


1.  In  how  many  possible  ways  could  the  experts  be  distributed  over  the 
cells  In  a  given  row?  Thus,  for  Row  1  of  Table  A-2,  expert  1  might 
have  had  three  performances  and  expert  2  one  performance,  or  expert  1 
three  and  expert  3  one,  and  so  on. 

2.  In  how  many  ways  could  each  expert  have  had  the  number  of 
performances  he  got?  E.g.,  If  an  expert  had  two  performances ,  and 
there  are  three  possible  performances  (Intuitive,  quasl-ratlonal ,  and 
analytical),  then  there  are  three  possible  pairs  of  performances  that 
he  could  have  had.  There  Is  only  one  way  he  could  have  had  three 
performances. 


Question  1.  This  question  has  to  do  with  the  Identity  of  experts. 
Calculating  all  the  possible  ways  of  distributing  C  *21  experts  over  the 
cells  of  a  given  row  is  done  by  multiplying  together  the  answers  to  a  sequence 
of  questions,  one  question  per  column.  The  first  question  Is:  In  how  many 
ways  could  rQ  experts  have  had  0  performances?  The  answer  Is 


l.e.,  211 

CrQ>!  *  (21  -  r0)!  , 


the  number  of  possible  combinations  of  21  experts  taken  rQ  at  a  time.  The 
second  question  considers  the  remaining  experts.  How  many  ways  could  r^ 
experts  out  of  (21  -  rQ)  have  1  performance?  The  answer  Is: 
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21  -  r„ 


This  process  Is  done  repeatedly  until  there  Is 


or  1 


21  -  rO  -  rl  -  r2  \  „  /  r3 
'3  /  v  r3 

way  of  assigning  the  remaining  r^  experts  to  the  last  column  In  the  row. 
These  binomial  coefficient  terms  are  multiplied  together  to  produce  a  term 
expressing  the  number  of  possible  ways  that  the  C  *  21  experts  could  be 
assigned  to  the  sets  of  size  1  In  the  pattern  expressed  by  the  row's  r^ 
values.  Generally: 

/  „  \ 

r  V1 


C 

ir 

j=i 


j-i 

C  -  i  r, 
1*0 


rj 


/ 


Question  2.  This  question  has  to  do  with  the  Identity  of  performances 
within  experts.  That  Is,  In  how  many  ways  could  each  expert  have  his  given 
number  of  performances?  Each  expert's  assigned  number  of  performances  are 
drawn  from  a  universe  of  three:  the  Intuitive,  quasl-ratlonal ,  and  analytical* 
performances.  There  are  3  ways  he  could  have  1  of  them,  3  ways  he  could  have 
2  of  them,  but  only  one  way  he  could  have  none  of  them  and  only  one  way  he 
could  have  all  three  of  them.  Generally,  there  are 


(0 


possible  ways  the  expert  could  have  1  performances  In  a  cluster  If  the  maximum 


number  of  performances  possible  per  expert  Is  E.  The  product  of  these  terms 
for  each  of  the  21  experts  Is  the  number  of  possible  ways  for  particular 
performances  to  be  distributed  to  the  experts,  given  the  pattern  of 
concentration  expressed  by  the  row. 


Multiplying  together  these  two  expressions,  representing  the  number  of  ways 
that  the  C  *  21  particular  experts  could  be  assigned  to  the  given  collection 
of  set  sizes  and  the  number  of  ways  that  the  E  *  3  particular  performances 
could  be  assigned  to  each  expert,  produces  the  total  number  of  ways  that  the 
pattern  of  distributions  represented  by  a  row  could  be  obtained. 


Step  4.  Determining  the  position  of  the  observed  pattern  In  Table  A-2. 

In  the  present  study,  the  group  of  outliers  that  had  only  4  members  had 
the  pattern  corresponding  to  Row  4  of  Table  A-2. 


Step  5.  Calculating  the  probability  of  a  given  row. 


The  previous  step  calculated  the  number  of  possible  ways  that  a  row'# 
pattern  could  occur.  Dividing  It  by  the  total  number  of  ways  that  all 
possible  patterns  could  occur  gives  the  probability  of  a  row  of  exactly  this 
pattern  of  distribution,  or  concentratedness.  The  total  Is  given  by  the  sum 
of  the  number  of  possible  ways  all  patterns  could  be  obtained.  It  Is  also 
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21  *  3 

4 

l.e.,  the  binomial  coefficient  of  the  number  of  Items  In  the  universe  (number 
of  experts  times  number  of  performances  per  expert)  over  the  number  of  Items 
In  the  cluster.  Thus  the  probability  of  getting  exactly  the  pattern  observed 
In  the  example  Is  .81. 

Summing  the  rows  up  to  and  Including  the  observed  row  give  the 
probability  that  a  pattern  this  concentrated  or  more  could  occur,  1.0. 
Sunning  the  probabilities  for  all  rows  from  the  observed  row  to  the  bottom  of 
the  table  gives  the  probability  that  a  pattern  this  concentrated  or  less  would 
occur,  .81. 

Table  A-4  shows  the  number  of  possible  ways  each  row  could  occur  and  the 
probabilities  of  observing  a  pattern  with  a  concentratedness  equal,  greater 
than  or  equal,  or  less  than  or  equal  to  the  given  row. 


Table  A-4 


Num.  of  possible 

Probability 

Probability 

Probability 

ways  the  pattern 

of  that  row 

of  the  row 

of  the  row 

could  occur 

or  less 

or  more 

Row  1 

1260 

.002 

1.000 

.002 

Row  2 

1890 

.003 

.997 

.005 

Row  3 

107730 

.181 

.995 

.186 

Row  4 

484785 

.814 

.814 

1.000 

Total 


595665 


1.000 
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